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ABSTRACT 



A one or two player virtual reality game efficiently 
detects and tracks a distinctively colored glove. Ac- 
cording to the preferred basketball embodiment, a sin- 
gle player equipped with the distinctively colored glove 
is matched up against a virtual opponent The object of 
the game is for the real player to put a virtual basketball 
into a virtual basketball hoop before his/her virtual 
opponent steals the ball. Initially, the background site is 
scanned, and then the operator with the glove is 
scanned. A table, of colors is then established which are 
unique only to the glove. A player is then scanned, 
against the background to identify which color glove 
will have the least conflict with colors worn by the 
player. During play, the player is scanned at 30 frames 
a second and the information is stored in a frame buffer. 
A prediction is made of the location of the glove in 
subsequent frames based upon its previously known 
location and velocity. Therefore, a search for the glove 
can be made of only a limited portion of the full frame, 
thereby increasing the speed of acquisition. Gestures 
made by the player such as a flick shot, a dribble or a 
roundhouse shot can be distinguished so as to automati- 
cally cause the basketball to be released from the play- 
er's hand. If the velocity and direction of the ball are 
substantially in the direction of the virtual basketball 
hoop, then the player will be credited with a score. 

15 Claims, 16 Drawing Sheets 
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lation: 4,590,469; 4,678,338; 4,797,738; 4,917,500; 
VIRTUAL REALITY GAME METHOD AND 4,954,972; 5,027,195 and 5,117,101. 

APPARATUS Tracking objects using a computer can be found in a 

number of contexts, the most common of which is in the 
BACKGROUND OF THE INVENTION 5 area of tracking missiles or airplanes with radar. Some 

1. Field of the Invention us - Patents discuss tracking in other contexts. See, for 
In general, the invention relates to a virtual reality example, U.S. Pat No. 5,173,865 which includes a 

game method and apparatus and, in particular, to a three-dimensional motion parameter estimator. The 
virtual reality game in which a player wears a distinc- system described in U.S. Pat No. 5,231,483 tracks a 
tively colored glove which is efficiently identified and 10 human being and produces an error signal based upon 
tracked during the course of the game. the individual's motion. That helps a camera motor 

2. Description of Related Art anticipate the future position of the object being 
The term 'Virtual reality" has come to mean the tracked. See also, U.S. Pat No. 5,227,985, previously 

superposition of a first image, typically a "real image" discussed, FIG. 9. 

against or in a second image, typically a background, to 15 Centroid determination in a context of tracking sys- 
create the illusion that the first image is, in fact, located terns appears to be discussed in at least the following 
in the second image. There are two types of virtual U.S. Pat Nos.: 4,958,224 and 5,175,694. 
reality, namely first person and third person. First per- Lastly, U.S. Pat No. 4,984,179 discloses a helmet 
son involves the participant directly whereas third per- used in the context of a virtual model but otherwise 
son allows the participant to see him or himself on a 20 does not appear to be as relevant as some of the other 
display in a specific environment The present invention references discussed above. 

relates to a third person virtual reality technology. As- Finally, the May 1992 issue of "Presentation Devel- 
pects of virtual reality have been known for some time. opment and Delivery" includes an article entitled "In- 
For example. Heads Up" displays are known m the teracting in a Virtual World" on pages 14-15 which 
context of aviation wherein a target, for example, might 25 describe$ a virtual reality ^0^^ which a user, 
be supermiposedupon the clear canopy of anairplane. wearing a ^lor gi ove , has his or her image superim- 

Also, certain aspects of virtual reality have been devel- ™«« * ^JT™,.^ — . ^ , , *, v , 

oped in the remote control art It is known, for example, ^J?™ tl ? ™? 
how to remotely grip a handle and manipulate a remote * C J^* * S^J^T? °^^ ul f n S 
object at some See. Such systems hive found early 30 *■£ » loca *™ Wy- ™ e object 

use in the nuclear testing and electrical generating field ****** ? to t0 J io ? 2* mcoi T£ ob ^ t 

where the handling of radioactive isotopes is required. 7^ or her ^ deflecting it from the 

Virtual reality is mentioned or discussed in at least the A computer tracks the number of successful 

following U.S. Pat. Nos.: D338,051; .D338.G50; blocks and displays a final score based upon the success 
5,227,985; 5,214,615; 5,193,000, 5,187,540, 5,185,561; 35 f ™ »«■ ^ «* sv stem just described has some 
5,184,319; 5,177,872; 5,165,897; 5,148,310, 5,148,154; features in common with regard to the present inven- 
5 142,506 and 5,130,794. tl0n * nevertheless, is a much more primitive and limited 

' Perhaps the most relevant of. the foregoing group is version and is incapable of performing relatively sophis- 
U.S. Pat No. 5,227,985 which discusses the use of a ticated *»* realistic interactive actions, 
camera for the purpose of determining the orientation 40 ^ was m ^ context of the foregoing prior art that 
of an object Also possibly relevant is U.S. Pat No. present invention arose. 

5,214,615 which discloses a tracking technique based SUMMARY OF THE INVENTION 

upon responses from a microphone. 

A body of technology also exists for the purpose of Briefly described, the invention comprises a virtual 
detecting and distinguishing between colors. For exam- 45 reality game apparatus and method which employs a 
pie, see U.S. Pat No. 4,488,245 entitled METHOD gto ve whose distinctive color is identified, tracked and 
AND MEANS FOR COLOR DETECTION AND predicted so that it can be employed as part of a sophis- 
MODIFICATION. That patent describes a method ticated interactive game. The preferred embodiment of 
and means for color detection and modification wherein the invention is known as "Jump Shod " or "Virtual 
the recognition of color is based upon the definition of 50 Hoops" and requires a player, wearing a special glove, 
a three-dimensional color sub-space within a three-di- to compete against a virtual opponent in a simulated 
mensional color universe. Initially, an object is scanned one-on-one basketball game. Prior to all game playing, 
and its chromaticity values are digitized and modeled as the system is taught to recognize each of several uni- 
a function of chrominance and luminance. Three 8-bit formly colored cotton gloves which will be used for 
words are stored for each pixel of the digitized color 55 tracking during game play. The result of the training is 
picture. The first value is an approximation of lumi- & set of master color tables in a thiee^limensional color 
nance, while the second and third values represent vari- scheme which classify all possible color values as glove 
ations of chromaticity or chrominance. The technique color or non-glove color. Colors in these tables are 
described can be used in robotic devices or in three-di- referenced with three 6-bit values, the chroma x, 
mensional physical space and specifically may be em- 60 chroma y, and luma value. The system establishes these 
ployed with computer-generated animation. . tables by the following procedure: First, the system 

U.S. Pat Nos. 5,012,431 and 5,136,519 describe color scans a background site to determine the colors present 
scanning techniques which employ a computer to iden- therein. All background colors are classified as non- . 
tify whether a particular color is present in a textile or glove colors. The system then scans the background site 
other context 65 with the operator present All colors present in this 

The following patents are cited as showing the gen- image are also classified as non-glove colors. The sys- 
eral state of the art of color recognition and detection tern then repeats the following process for each glove 
technology, but are not offered as ah exhaustive compi- color it learns to recognize: first the operator puts on a 
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glove of the color to be learned. The operator is of the full frame buffer for the glove. The limited search 
prompted to place his or her gloved hand in five to ten is performed in a coarse-to-fine manner starting at the 
different boxes on the video screen so that the system predicted position of the glove and radiating outward 
can scan the image of the glove in various different Initially, every fourth line of the frame buffer is search, 
positions and flexures. All pixels scanned within the 5 then every other line. If this limited search fails, a 
boxes are classified, as possible glove colors and their coarse full-buffer search is performed in which every 
frequencies of occurrence are tabulated. The system sixteenth line is searched. Failing the full buffer search, 
pares down the table of possible glove colors to a table the current field is discarded and another acquired For 
of actual glove colors by removing from the possible the next digitized field, a search is performed in an area 
glove color tableau colore which have previously been 10 larger than that before, but still smaller than the full size 
classed as non-glove colors. of the frame buffer. This search begins at the last known 

At the begmning of each game the player is scanned position of the glove. This search is again coarse-to-fine, 
standing against the background. All colors found in searching first every eighth line, then every fourth. If 
this image are classified as non-glove colors. The play- this limited search fails, a coarse search of the full frame 
er*s clothing and skm colors are compared with the 15 buffer is performed. If the full buffer search fails a sec- 
tables of known glove colors established above. A color ond time, the second field is discarded and a third ac- 
of glove is chosen which maximizes the frequency of quired. The full buffer is searched in a coarse-to-fine 
the glove colors in the table which do not conflict with manner starting at the center of the screen and radiating 
colors found in the image of the player. The player then outwards. First every sixteenth line is searched, then 
puts on a glove of the color selected. The system cus- 20 every eighth, and then every fourth. New fields are 
tomizes its table of known glove colors to the current acquired and the full-buffer is searched in this manner 
player by performing a brief training session. The until the glove is found. 

player is prompted to touch four or five objects on Throughout the search process, pixels from the frame 
screen, and the player is scanned each time he or she buffer are compared to the table of known glove colors 
touches one of the objects. As above, all pixels scanned 25 for the selected glove. Each pixel in the frame buffer 
near the objects are classified as possible glove colors. has its chroma and luma values spread out over 8 bytes 
Also, all colors which were previously classified as in the frame buffer. Rwmiirfng all 8 bytes is a computa- 
glove colors for the selected color are added to the table tionally expensive process. Since enough information is 
of possible glove colors. The system pares down the contained in the first 2 bytes of these 8 bytes to rule out 
table of possible glove colors to a table of actual glove 30 most pixels from being glove colors, a second, abbrevi- 
colors by removing from the possible glove color table ated table of glove colors is established which allows 
all colors which have previously been classed as non- the system to classify most pixels as non-glove color by 
glove colors. examining only the first 2 of 8 bytes. Pixels which can- 

The play of the game is typically made against a not be classified by the initial table then have all 8 bytes 
60-second shot clock. The player initially touches one 35 checked. 

of the basketballs and makes a dribbling motion. The When the glove is located, its centroid is determined 
virtual opponent then comes out to try to block the in order to assign consistently to the glove a single (x,y) 
player. If the virtual opponent touches the basketball, value. This value is transmitted from the computer 
then the virtual opponent scores a steal and slam dunks which performs the tracking to the computer which 
the ball through the basket, thereby giving the virtual 40 runs the game via a serial line, 
opponent a score. The real player, whose image is su- Other useful features of the present invention include 
perimposed on a display with the virtual player, can the ability to automatically determine the height of the 
shoot the basket by making one of three distinctive player and scale the height of the virtual opponent in 
gestures including a flick shot, a dribble shot and an proportion to the size of the player. The width and the 
overhand throw. The system can recognize these mo- 45 outline of the player are also acquirable and can be used 
tions as three distinct gestures and interpret them as for purposes of detennining when the player bumps into 
instructions for the simulated basketball to leave the objects or the side frame of the background, 
hand of the player. If the direction and velocity of the The basic concepts can be employed for other types 
hand is substantially in the direction of the virtual bas- of games, For example, it is possible to play a two 
ketball hoopoe, the goal, then the ball will appear to go 50 player game in which two real players each wear gloves 
through the basket and the player will get two points. If of different distinctive colors and compete with each 
the real player makes more points then the virtual oppo- other on a one-on-one basis to put the basketball into a 
nent during the 60 seconds of play, then the real player virtual hoop. Another embodiment permits the player 

wm f* to use his or her hand to move the player's whole body. 

The system requires the position of the player's 55 For example, the player could be superimposed upon a 

gloved hand throughout the game. Glove position is moonscape and the movement of the gloved hand 

determined by locating collections of glove-colored would cause the entire image of the player to jump uo 

pixels in the scanned image of the player. The system and down as if in an environment of substantially re- 

digitizes the video image of the player 30 times per duced gravity. 

second and stores the digitized video fields in a frame 60 These and other features of the invention may be 

buffer. Each field is approximately 640X240 pixels. more fully understood by reference to the following 

This is a large amount of date to search for the glove, drawings, 

and since the game requires glove position data in real ' 

time, the time required to locate the glove must be BRIEF DESCRIPTION OF THE DRAWINGS 

small Accordingly, the system includes a method for 65 FIG. 1 illustrates the preferred embodiment of the 

predicting the future location of the glove based on the invention in which a real player, wearing a glove hav^ 

glove's last known location and velocity. The system ing a distinctive color, plays a one-on-one basketball 

will therefore normally search a limited, smaller portion game against a virtual opponent 
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FIG. 2A is a schematic of the video elements that such as basketball 18 into a imaginary, virtual basketball 

comprise the preferred embodiment of the invention hoop 22 supported by a virtual backboard 24. To make 

illustrated in FIG. L the game 10 more realistic, the player 14 has to shoot 

. FIG. 2B is a schematic of the audio elements that over a virtual opponent 20. If player 14 does not shoot 

comprise the preferred embodiment illustrated in FIG. 5 the basketball 18 in time, the virtual opponent 20 may 

1. steal the ball from Mm and slam dunk it into the virtual 

FIGS. 2C and 2D illustrate other related electronic basket 22. Also, player 14 may lose the virtual basket- 
schematics included in the preferred embodiment of the ball 18 if the trajectory and velocity of the ball 18 are 
invention illustrated in FIG. L not such as to substantially bring it into the vicinity of 

FIG. 3 illustrates an alternative embodiment of the 10 the virtual basketball hoop 22. Each time player 14 

invention in which two real players, wearing different makes a successful shot, the score board 26 will give 

distinctly colored gloves, play one-on-one against each him or her an appropriate point Conversely, if the 

other in a virtual basketball setting. virtual opponent 20 is successful in stealing the ball 18 

FIG. 4 is an electronic schematic of the alternative and making a basket, the opponent 20 receives a score, 

embodiment of the invention illustrated in FIG. 3. 15 Typically, the game 10 times out after about 60 seconds 

FIG. 5 is a global flow chart illustrating the overall and the player 14 or 20 with the highest score, as shown 

invention. ' on scoreboard 26, wins. The basketball environment 12 

FIG. 6A and 6B illustrate the method steps for creat- can also include a variety of other visual props such as 

ing the master color table. grandstands 28 rilled with virtual fans. 

FIG. 7 illustrates the method steps for calibrating the 20 FIGS. 2A-2D schematically illustrate the electronics 

system for a specific player and a specific glove. and hardware that implement the preferred embodi- 

FIG. 8 is a flow chart illustrating the search steps ment of invention 10 illustrated in FIG. L The manner 

performed by the tracking software, of operation of the electronics illustrated in FIGS. 

FIG. 9A illustrates the first stage of a predictive posi- 2A-2D is explained in further detail in FIGS. 5-13C, 

tion search. 25 which are discussed subsequently in this disclosure. 

FIG. 9B illustrates the second stage of the predictive FIG. 2A illustrates the video wiring schematic 30 of 
search if the glove is not located during the search of the preferred embodiment 10. A ^camera 32 scans the 

the smaller search area illustrated in FIG. 9 A. pla yer 14 and produces an outputJo.Matr6TJ4rtramera 

FIG. 9C illustrates the coarse-to-fine technique em- 32 also interacts with SupeT Video Windows card 36 

ployed to expeditiously search the predicted glove area 30 wm^E^^ToT^e^^^^g_computer system. Matrox 

shown in FIGS. 9A and 9B above. 34 also receives an input from LDV 8000 l aser disk 3 8 

FIG. 10 illustrates the technique employed for locat- ancTproduces an outpurf6Tlfie"Kfa0ai~video overlay 

ing the centroid of the glove. board 40 which in turn drives monitor 42. Monitor 42 

FIG. 11 illustrates a typical frame of 640 x 480 pixels preferably comprises a multi TV large screen display in 

as stored in a frame buffer. 35 which nine large catherode ray tubes are organized in a 

FIG. 12 illustrates the manner in which the chroma three row by three column array. Other types of prior 

X, chroma Y and luma information are stored within art large screen displays would also be appropriate for 

the frame buffer with the most significant bits of the the invention. 

chroma X and chroma Y values stored in the first byte The audio wiring schematic 50 of the preferred em- 

of the 8 byte sequence, and a luma value is stored in the 40 bodiment 10 is illustrated in FIG. 2B. Laser disk 38 also 

second byte. provides an audio output to a Gravis ultrasound unit 5Z 

FIG. 13A illustrates a sample possible glove color Ultrasound unit 52 provides an input to audio amplifier 

table and a sample non-glove color table. 54 which in turn drives speakers 56. 

FIG. 13B illustrates the initial separation of glove FIG. 2C illustrates the pin connection arrangement 

colors from the table of possible glove colors. 45 schematic 60 of the preferred embodiment 10. Game 

FIG. 13C illustrates the process by which spuriously computer 62, which is understood to include Matrox 34 

classified pixels are eliminated. and Magni video overlay board 40, both shown in FIG. 

FIG. 13D demonstrates the effect of a region-grow 2A, is connected via digiboard 64 to the digiboard four 

operation on a table of glove color values and on a table port breakout 25 pin male connector 66. The nine male 

of non-glove color values. 50 pin 68 of the tracking computer is connected via the null 

FIG. 13E illustrates the process of removing ambigu- pin 70 to connector 66. Similarly, the button control box 
ous color values from the region grown tables of glove is connected through twenty-five pin-mod 72 to con- 
color values and non-glove color values. nector 66. Laser disk 38 is connected to connector 66 in 

DETAILED DESCRIPTION OF THE « ^ ^ 'I^V?^ mG : 20 a J. nst ?J tes decbonic 

PREFERRED ]EMB ODIMENT 55 f^f. 80 ^LfT ^ ^ M ^ Z"^ 

nu.iiJMu-u wTiuwii^i board 82 is connected to the slider control box 84. 

During the course of this description, like numbers A two player alternative embodiment 100 is iUus- 
are used to identify like elements according to the dif- trated in FIG. 3. The alternative embodiment 100 is 
ferent figures that illustrate the invention. played in a basketball stadium 12 using a first human 

The preferred embodiment of the invention 10 is 60 player 14 who wears a glove of a distinctive color 16 
illustrated in FIG. 1. A basketball court 12 is shown just like in the preferred embodiment 10 illustrated in 
over which the image of a real, human player 14 is FIG. L As before, the object of the game is to place the 
superimposed. Player 14 wears a distinctively colored virtual basketball 18 through the virtual hoop 22 at- 
glove 16 on either the right or left hand. Glove 16 is tached to the virtual backboard 24. Score is kept on the 
chosen so that its colors do not conflict with those of 65 scoreboard 26. Unlike the preferred embodiment 10, 
the player 14 or the normal background against which however, the alternative embodiment 100 contemplates 
the player appears. The object of the preferred embodi- a second human player 102 instead of the virtual oppo- 
ment 10 is for the player 14 to shoot a virtual object, nent 20 illustrated in FIG. 1. The second human player 
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102 also wears a distinctive glove 104. The distinctive 
color of glove 104 is different from the color of glove 
16. Both gloves, however, have colors that are not 
found, either on the individuals 14 or 102, or on the 
background against which their images are scanned. ^5 UH 

The alternative embodiment 100 is played similar to the 1 initialized, the game 10 enters into its main 'loop where 



are illustrated in FIGS. 6A-6B. It is generally only 
necessary to do a master color table calibration when- 
ever the site is changed- Next, it is necessary to perform 
a color calibration for the participant as illustrated in 
step 500 and as shown in FIG. 7. Once the system 300 is 



?! C /^™™^ m ^!i^ gggl^ff?" P^ 14 \ it tracks the glove until the simulation is over as fllus- 

' * trated by step 600 and summarized in FIG. 8. 

The overall steps 300 are performed by preferred 
embodiment 10 with the following elements: 
486DX2/66 LOCAL BUS (Game Computer) 
Hard Drive 

Actix Graphics Engine 32VL+2Mb SVGA. 
Magni Video Overlay Board 
Matrox Illuminator Pro 
Gravis Ultrasound 
Digiboard 4 port 
486DX/33 (Tracking Computer) 
Hard drive (40Mb min.) 
VGA card 

Super Video Windows card 
LDV-8000 LASERDISC PLAYER 
SONY VIDEO CAMERA 



10 



IS 



or 102 can steal the virtual basketball 18 from the other 
and put it through the goal 22. 
v FIG. 4 is an electronic and mtxrhnmnn] schematic 200 
/of the system behind the alternative embodiment 100 
illustrated in FIG. 3. Each of the players 14 and 102 are 
scanned against a backlit Chromakey background 202 
and 204, respectively. The image of player 14 is cap- 
tured by a first imaging camera 208 and the image of the 
second player 102 is captured by a second 
camera 212. In addition, a first trpcVtn g camera 2 06 
tracks the glove 16 of the first player 14 and a second 
trac^gjcmerajlO tracks the glove 104 of the second 
player 102. The output from the first tracking camera 20 
206 is fed.to a first tracking computer 214 which in turn 
feeds ga me computer 218. Similarly, the second track- 
ing camera 210 feeds a second tracking computer 216 

which also provides an ad mtional Jnput to the game c^vrv Aim^ * ^ 

computer 218. The first andle^a^a^g^Seras 25 ^^1° AMPLIFIER 
208 and212 receive their svnc signal from sync genera- 
tor 238. The outputs, respectively, from the first and 
second imaging cameras 208 and 212 go to a first and 
second Chromakeyer unit 234 and 226, respectively. 
Another portion of their signals is transmitted through 30 
delay lines 236 and 228 and separated as red, green and 
blue (R,G,B) components as additional inputs to the 
first and second Chromakeyer units 234 and 226. The 
outputs from the first and second Chromakeyer units 
and 234 and 226 provide inputs, through time based 35 
correctors 232 and 231, respectively, to a first and sec- 
ond switcher card 224 and 222 of the video computer 
220. The background input from the second 
Chromakeyer unit 226 provides an input to distribution 
amplifier 230 which also rec eives ag _ input ftpin.lager 40 
disk player 246 frrnnflh trnw hacprf corrector 233. The 
output from distribution amplifier 230 provides a sec- 
ond input to the first switcher card 224. The output 
from the first switcher card 224 also provides an input 
to the second switcher card 222. The output from the 45 
second switcher card 222 provides an additional input 
to the game computer 218. Game computer 218 pro- 
vides for the overlay of the images of players 14 and 102 
against the background and also provides the video wall 
control signal. The output from ga me computer^ 218 50 
provides an input Jo^video w all processor 240 whicn m ] 
turn drives the nine CRT cube video wall 24 2. <^ 

A~third possible embodiment of the invention is the 
simulation of near weightlessness against a moon-like 



2 SPEAKERS 
VGA MONITOR 
CABLES 
BNC cables 
9 pin to RGB 

muminator Pro input cable & 2 BNC adaptors 
murninator Pro output cable 
RCA cables 
. Speaker wire 

25F to 9F with null modem 
4 wire modular 
25F to Laserdisc 15M 
9M to 9F (long) 

Magni Modular cable (8 conductor) 
Power Cables 
VGA extension cables 
Keyboard extension 
MISC. 
Keyboard 
VGA terminator 
RCA to mini adaptors 
Digiboard 4 port breakout box 
25F to modular (4 conductor) 
Camera Power Supply 
Magni Control Box 
2 Mice 

The initial training of the system 10 enables it to 
recognize a set of glove colors. The glove colors are 
learned by using the following procedure after the start, 



backdrop. A single individual, though there could be 55 illustrated by step 402, has begun. 



more than one, has the identifiable color glove on one 
hand. The system recognizes movements of the glove 
and, therefore, if the moon walker makes a rapid up- 
ward or downward movement, the system will take his 
or her entire image and move it a significant distance 60 
over the moonscape, thereby simulating walking a low 
gravity environment 

FIG. 5 illustrates a global flow chart 300 of the basic 
tracking system. After the start 302, the system asks if 
this is the first time the program has been run at a new 65 
site as illustrated by step box 304. If this is the first time 
it has been run at that site, then the system performs a 
master color table calibration, the steps of which 400 



First, the background is scanned 4 or 5 times, and a 
table of the colors present is constructed. All color 
values found in this table are classified as non-glove 
colors. See step 404 of FIG. 6A. In addition, this table 
is stored as a table of background colors for use with 
subsequent height/width measurement and/or outline 
tracking. 

Next, a number of images of the system operator, 
while wearing neutral-colored clothing, are scanned. A 
table of all of the color values found in these images of 
the operator is constructed Any color value in this 
table is identified as a non-glove color. See steps 404 and 
406 of FIG. 6A. 
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Next, the operator dons the colored glove which he shown in step 512, until the hand reaches a preselected 
or she wishes to train the system to recognize as illus- region. As shown in step 514, the player or participant 
trated by steps 408 and 410. A box then appears in a is scanned again and the pixels outside a box surround- 
preselected position on the screen, as illustrated by step ing the glove are identified as non-glove colors, and 
412, and the operator then places his hand in such a 5 pixels inside the box are identified as possible glove 
fashion that it appears in the box on the video screen as colors. The process is repeated several times, as shown 
shown by step 414. The operator keeps his hand in the by step 516, after which the glove colors are separated 
box until prompted to move it Tables are constructed from non-glove colors and further post-processing is 
of all the color values present inside the box containing performed. Once step 518 is completed, the participant 
the hand and of all the colors present outside the box. In 10 or player calibration process 500 returns with a new 
addition, the frequencies of occurrence of color values color table tailored to the player or participant who will 
inside the box are tabulated. The colors outside the box play the game, as shown by final step 520. 
are identified as background colors. The box contains In summary, for each glove color that the system has 
both the gloved hand and some background pixels, so been trained to recognize, the amount of conflict is 
the colors inside the box are identified as possible glove IS computed between the glove's color and the back- 
colors. The procedure is repeated several times, as illus- ground table as described above. A glove color is 
trated by step 416, with the box located in different chosen which maximizes the average number of pixels 
places so that both hand and operator are seen from a in the glove which do not conflict with the background, 
variety of angles. Glove colors are then separated from Once the glove color is determined, according to the 
non-glove colors in step 418 and post processing is 20 process 500 illustrated in FIG. 7, the table of glove 
done, thereby creating a preliminary glove color table. colors is refined by the following procedures. 

The second stage in the generation of the master Initially, the player dons the colored glove which has 
color table is illustrated by steps 420-428. Theinforma- been selected by the system 500. The system places 
tion in the color table is further refined to improve its virtual objects on the screen which the player must 
quality. A box, as suggested by step 420, appears on the 25 touch. Using the master table of glove colors from the 
screen in a preselected position. The operator's glove is initial training, the system is able to automatically deter- 
then tracked using the color table generated by steps mine when the gloved hand touches the objects. When 
402-416 until the hand reaches a preselected position as the hand enters a box, which is not visible to the person, 
shown by step 422. The operator is then scanned again but which surrounds the objects to be touched, tables of 
such that pixels outside of a box surrounding the glove 30 possible glove colors and non-glove colors are con- 
are classified as non-glove colors and pixels inside the structed in the same way as in the initial training session, 
box are classified as possible glove colors and their This procedure is repeated several times with the ob- 
frequencies of occurrence are tabulated, as illustrated in jects located in different places so that both hand and 
step 424. The process is repeated several times, as illus- person are seen from a variety of angles, 
trated by step 426. Next, glove colors are separated 35 A new refined table of glove colors is constructed by 
from non-glove colors and further post processing is . combining the master glove color table for the selected 
performed. The refined color table is then saved to disk. glove color with the current table of possible glove 
Once this is done with the first colored glove, then the colors found inside the boxes, and then removing all 
process is repeated with all other colored gloves, as colors currently classified as non-glove colors. Spurious 
illustrated by step 430, until each different color of 40 pixels are removed and gaps in the table are filled in 
glove has had a master table of glove colors created for during subsequent post-processing. This new table of 
it At this point, the initialization is concluded as illus- glove colors is used for tracking the hand during the 
trated by step 432. game. The post-processing operation is further de- 

The ultimate goal of the generation of the master scribed below, 
color tables illustrated in FIGS. 6A and 6B is to create, 45 In order to understand the problem, it is important to 
for each different colored glove, a table of colors found know that during calibration, two tables are created, to 
exclusively in that color glove. For each glove, the final wit 

table is constructed by taking the table of possible glove 1. A table of non-glove colors present on the person 
colors found inside the boxes and removing all colors and in the background; and, 

previously classified as non-glove colors. Gaps in the 50 2. A table of colors which are possible glove colors, 
tables are filled in and spurious pixels are removed dur- From these two tables, it is necessary to create a 
ing subsequent post-processing. The final table is saved single table of glove colors which are not duplicated in 
to disk as is the table of the frequencies of occurrence of the participant's clothing, skin, or in the background, 
the pixels in the final color table. The entire procedure Le. color values which are unique to the glove, 
of generating these master color tables is performed 55 What is done is the following: 
once per site. A color is a glove color if (1) the color is found in the 

The next process is to determine which glove the table of possible glove colors; and, (2) the color is not 
participant or player 14 should wear. That process 500, found in the non-glove table. 

which involves calibration of the participant, is illus- The foregoing process is not as simple as it may ap- 
trated in the steps shown in FIG. 7. After the start 502, 60 pear for two reasons. First, there is a very large set of 
the player is scanned to determine what colors he or she possible colors, over two million, which can be distin- 
is wearing, see step 504, and then a master table of glove guished by the digitizing board. Even on a glove which 
colors is read from disk and the player's clothing is is perfectly uniformly dyed to a single color, there may 
compared for color conflicts as shown in step 506. The be many thousands of possible color values which the 
color with the most frequent non-conflicting color val- 65 digitizing board recognizes, due to subtle differences in 
ues is selected,, as shown in step 508, as the optimal lighting and hand orientation, and due to edge effects 
glove color. Next, the player 14 puts on the selected from averaging the glove color with adjacent arm/shirt 
glove, as shown in step 510, and the glove is tracked, as pixels and/or background pixels. In a typical training 
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session, only a fraction of the possible glove colors and between neighbors of neighbors of observed glove col- 
background colors will be observed. The system for ors and observed non-glove colors, and so on. The 
classification will fail for pixels not directly observed or system thus compares data whose accuracy is compara- 
otherwise accounted for during training, so it is impor- ble at each step. 

tant to take measures to improve classification. Second, 5 In order to cope with the noise, which is the second 
there is some noise present in the system, so some pixels problem, it is necessary to make a similar assumption, 
will be spuriously misclassified. namely, that if we classify and observe color as "X," 

We wish to classify all color values as either glove then the system should also classify similar colors as 
color or non-glove color. In order to remedy the first "X." If we then find a color classified as a glove color, 
problem above, the following assumption is made: Sup- 10 yet all neighboring colors are classified as non-glove,' 
pose we have a color value C which is classified as "X" then it is reasonable to eliminate the glove-colored clas- 
(where "X" is either "glove color" or "non-glove sification of the isolated spot in the region. This process 
color"). Then all other color values C l with chroma and of elimination is illustrated in FIG. 13C The end result 
luma values sufficiently close to C will have the same is the table 966. 

classification "X." This assumption is clearly not true 15 The position of the hand of the player is determined 
for all color values— if it were, then by induction, all by locating collections of pixels in the digitized image of 
color values must have the same classification. How- the person to be tracked which have color values con- 
ever, it is a useful starting point tained in the table of glove colors. Because the image is 

The extraction of the table of glove colors from the quite large (over 300,000 pixels) and because quick 
table of possible glove colors is illustrated as process 20 search time is required to give real-time position infor- 
900 in FIGS. 13A-13E. A sample possible glove color mation, the search must be done in a highly efficient 
table 940 and non-glove color table 946 are illustrated in manner. The use of a digitizing board with a frame 
FIG. 13A. Note that this illustration is of a single luma buffer enables much more efficient searches, 
layer of the tables. All tables d iagramm ed here are rep- An overview of the tracking search technique 600 is 
resented as two dimensional slices of three dimensional 25 illustrated in FIG. 8. The process starts which search 
tables. All non-glove colors 948 are removed from the step 602. Thereafter, an initial prediction is made for the 
table of possible glove colors 940 as illustrated by pro- location of the hand and the region of the frame buffer 
cess 920 in FIG. 13B, and we are left with a table of to search as shown in step 604 and as further described 
observed glove colors 954 and a table of observed non- in FIGS. 9A and 9B. In the next step 606, an initial 
glove colors 946. The removal process proceeds in a 30 coarseness for the search is chosen as further described 
similar manner for all luma levels. with reference to FIG. 9C. A search is subsequently 

Fust it is desirable to eliminate any spurious pixels. performed, as shown by step608, and a determination is 
This procedure 960 is described in detail below and made to see if the gloved hand is found as shown by step 
illustrated in FIG. 13C. The denoised glove color table 610. If the gloved hand is not found, then the search 
966 becomes the new glove color table and is used by 35 refined in step 612, as also illustrated in FIG. 9C, and 
the gap-filling procedure which follows. another detennination is made to see if the hand is found 

Next it is desirable to fill in gaps in these tables. First, as shown by step 614. If the second determination 614 
it is useful to perform a region grow operation on the indicates that the hand is not found for the second time 
table of glove colors 966 and on the table of observed then the search is further refined or the whole frame is 
non-glove colors 946 as illustrated by process 930 in 40 searched as illustrated by step 616 and as is also illus- 
FIG. 13D. The operation "region grow" is defined on a trated in FIG. 9C. If after the third attempt the hand is 
color table T as follows: For each color value (X,Y,L) not located, in steps 616 and 618, then the system 600 is 
(where X,Y, and L are the color's chroma x, chroma y, told to stop as illustrated by step 626. If at any time, the 
and luma values, respectively) in T, all color values gloved hand is located in steps 610, 614 or 618, then the 
^nLOsuchthatdther(l)L=Liandmax(|X-X>|, 45 upper and lower limits of the hand are identified in steo 
| Y-Y1|)=1, or (2) |L-LM =1 and X=X> and Y=Y» 620. Thereafter, the centroid of the hand is determined 
are added to the table. The region grow operation adds by the technique illustrated in step 622, as shown in 
to each table all the neighbors of each pixel in the table. FIG. 10 and as described in detail below. Subsequently 
By our assumption, neighbors of a color in a table T the system returns the position of the hand as shown by 
belong to the same table T. Of course, our assumption 50 step 624 and, after the game 10 is completed, the track- 
does not always hold true, and there will be colors ing system comes to a full stop as shown by step 626 
which are neighbors of both glove colors and non-glove,,^ The frame buffer in the system enables video fields to 
colors. Such colors are deemed to be of ambiguous x be searched ratherj han searching full video frames, a 
classification, and they are discarded from both tables as tecWqiie-wiu^spe^^ The 
illustrated by process 980 in FIG. 13E. The process of 55 time required to acquire a field is 15 milliseconds, rather 
filling gaps as illustrated in FIGS. 13D and 13E is re- than the 30 milliseconds requkedto acquire a full frame, 
peated one to two times. The end result is the smoothed A typical frame 850 is illustrated in FIG. 11. A conven- 
taWe of glove colors 992. tional frame 850 is 640 pixels wide by 480 pixels high. 

The process occurs in several stages because at each Therefore, a full frame is considered to be 640X480 
step the chances increase that the assumptions about the 60 pixels. A "field" is defined in this case as half of a frame, 
region-grown colors are false increase. In other words, consisting either of only odd horizontal lines (an odd 
the classification of initially observed colors is probably field) or only even horiz ontal lin es (an even field) 
correct The classification of the neighbors of the ob- Because the motion of the hand is continuous, and 
served colors is less accurate, and the classification of because the tracking system obtains frequent position 
the neighbors of the neighbors of the observed colors is 65 data, th e position of the hand from one field to thenexT 
more suspect The separations are performed in stages can be predicted fairly accurately. By first searching 
so that it is possible to see conflicts between observed near the predicted position of the hand, searchtime can 
glove and observed non-glove colors rather than, say, be cut dramatically. Depending upon ttTamoum of 
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previous data available, the predictive search is per- 
formed in one of three ways. 

First, as shown in FIG. 9 A, if the system has cor- 
rectly located the hand in both of the last two fields 
searched, then those last two positions are used to deter- 5 
mine the p osition and velocity of the hand in the previ- 
ous field, and the predicted position is obtained by as- 
suming that the velocity of the hand remains constant 
between the previous field and the current one. This 
prediction is accurate except when the hand accelerates 10 
or decelerates rapidly. As shown in FIG. 9 A, the first 
predictive technique notes the position at time ti as xi, 
yi, and t2 as x& yz Using this information, the system 
can fairly accurately predict the position of the hand 
during the next interval tp as being at point Xp, yp. Using 15 
this information, the system searches a relatively small 
field 702 rather than slowly processing a full frame of 
data. 

Second, if the system did not locate the hand during 
the last search, as illustrated in FIG. 9B, then the system 20 
predicts that the position of the hand will be the same as 
the last known position. This estimate is less accurate 
than when velocity data can be obtained, but it sail 
narrows the search considerably. Under this second 
scenario, a larger search field 70* is searched because of 25 
the higher likelihood of error in the predicted hand 
position. 

Third, if the software has not found the hand in the 
last field search, after the steps illustrated in FIGS. 9A 
and 9B, then the entire 640X480 frame buffer is 
searched. This is obviously much slower and time con- 
suming and ultimately creates a jerky motion on the 
video monitor 42 or 242. 

Because, in general, the image of the hand is large 
with respect to pixel size, it is usually unnecessary to 
search every single pixel in order to locate the hand. 
Initially, as shown in FIG. 9C, a coarse search is per- 
formed, scanning only every sixteenth line. If this 
search is unsuccessful, the search is refined, and every 
eighth line is searched. The process of refinement is 
continued until either the hand is located or until every 
line in the field 706 has been searched. 

In general, the image of the hand contains a large 
number of pixels. A single hand position must be deter- 4J 
mined in some consistent way from this large collection 
of pixel data. It has been learned that the left uppermost 
hand pixel, according to earlier attempts, was difficult 
to accurately locate. Since boundaries of the hand are 
irregular, it is hard to determine accurately the precise ^ 
location of the hand. However, according to the pre- 
ferred embodiment, the system is capable of detecting 
the centroid of the hand. The initial data 800, after 
classification, is illustrated in FIG. 10. The data includes 
glove pixels 802 and non-glove pixels 804. The bound- 55 
aries of the image are determined by a binary search for 
edges. It begins by starting at the first hand pixel located 
and performing the binary search for 2 lines in the frame 
buffer which are free of hand pixels. Once this is com- 
pleted, the centroid of all hand pixels within the bound- 
ary is computed by the following calculations: 

Centroid cx,cy 

Let S be the set of glove pixels inside the bounding 
box (see FIG. 10). Then 65 
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where n is the number of pixels in S. 

Another advantage and improvement of the present 
invention makes use of a unique and efficient coding 
scheme of the color table. The color values for each 
pixel are spread over eight bytes, Le. each 2-byte word 
contains four bits of chroma information and seven bits 
of luma information. The table of glove colors makes 
use of twelve bits of chroma information and six bits of 
luma. In order to verify that a pixel is in the glove, it is, 
therefore, necessary to check six bytes in the frame 
buffer. However, the vast majority of color values in a 
given image do not lie in the glove table. It is, therefore, 
important to determine if a pixel is a non-glove color 
with a minimum of effort It has been learned, fortu- 
nately, that the first 2 bytes of each pixel of color infor- 
mation contains enough chroma information to deter- 
mine for most pixels that the pixel is not of glove color. 
Therefore, a second glove color table is set up which is 
indexed by the first four bits of chroma information and 
the first six bits of luma information. If a pixel is found 
in the second table which requires only that two bytes 
be checked, then a full six-byte check is done using the 
first table. Since most candidates are eliminate! by the 
first two-byte check, this technique saves considerable 
search time. 

Another major unique advantage of the present in- 
vention is that it is capable of recognizing hand gestures 
so that the system knows when the player 14 intends to 
dribble or shoot a ball. The last ten observed hand posi- 
tions (x and y coordinates) are kept in an array. The 
following gestures are then recognized based upon this 
information: 

1. Roundhouse throw (overhand throw): Notation: 
The x,y cords of the hand at time t are stored for 
the last 10 observations. The current observation 
has index 0. The one before has index —1, and 
before that has —2, etc. So the current hand posi- 
tion and time is x(0), y(0), and t(0). Criteria for 
roundhouse throw: 

a) At least 9 samples must have been acquired 

b) The current velocity of the hand must exceed a 
. preset threshold 



W0) - <-2)l 



[*0) ~ X-2)] 
M0)-«-2)J 

c) The angle of the current direction of motion 
must have changed by more than a preset thresh- 
old. Thus, 

lea - ei\ > threshold. 

Here 02 - ttn-'RKP) - J<-2V(jc(0) - Jt(-2))] 
9l « tm-l[(K-4) - X-6)y«-4) _ x(-6))] 

2. Dribble (as with a basketball) Criteria for dribble: 
a) At least 9 samples must have been acquired. 
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b) The current velocity of the hand must exceed a Case 3: v is large (Le. v> =vl)— do no smoothing so 
preset threshold (this threshold is different from that response time is not slow 

that for a roundhouse throw) sx=x (0) 

sy=y(0) 

5 For some applications, it is useful to know the size of 

Here p « \(v£ + vy*) the rarticipant During the initial training stage, a table 

of all colors which appear in the background is con- 
where vx=[x(0>x(-2)]/[t(lW-2)] and structed. See FIG. 6A, 404. Upon receipt of the "mea- 

roaWflW M/rwm „ ,* , n mm arigto" command, the tracking system scans the 

^^-WWM-ffl to player md ^ imagc for ^ clo$est 

c) The angle of the current direction of motion P *^? 10 andto * e 1)0110111 

u ^ \S u awuoa c f ^ p^ye^g image. These pixel groupings are deter- 

have by more th^api^tthr^h- ^^be theloTand bot£m oftfS^re- 

old (again, this threshold is different from that ,„ — Al f . . - " *^ uwyam, ic- 

for aSdhouse throw). 15 spectively From the position of the top and bottom of 

; the participant m the image, a good estimate of the 

participant's height can be obtained. With this informa- 
Tto, |02 -6i I > threshold. tion, it is possible to automatically scale the height of 

Hoc oi - tan-i[(*0) _ *-2)/c*(Q) - *<-2)) P kyer or Participant 14 with respect to his virtual 

20 opponent 20 so that they are more equal in height and, 
Si = tan->((K-3) - j<-5))/(x<-3) - 4-5))] therefore, more lifelike. 

Similarly, in a number of applications, it is useful to 
3. Flick throw (for flicking an object off the hand know the approximate location of the player's body, 
with a quick wrist motion) Since detennimng the exact location of the edges of the 

Criteria for flick throw: 25 body is a very time-coiisuining task, the screen is 

a) At least 9 samples must have been acquired scanned until non-background pixels are found at least 

b) The current velocity of the hand must be less two separate rows. The average of the bounds on these 
than a preset threshold rows is taken to be a set of rough horizontal bounds on 

Here the position of the participant If one knows the width 

30 of the participant, it is possible to use that information 

. ( so that the participant can, for example, "bounce off the 

Here v = + yfi walls" or otherwise interact with other objects. Also, 

for many applications, it is useful to know when the 
where vx=[x(0>x(-2)]/[t(0>t(-2)] and vy=[y(0)- participant's body has touched a certain area on the 
y(-2)J/K0>t(-2)] 35 screen. With outline tracking, a number of rectangles on 

c) The velocity of the hand must have recently the screen are specified. The borders of these rectangles 
exceeded a preset threshold. are continuously checked for non-background pixels. 

where vxo=[x(-2>x(-4)] and vy 0 =[y(~2> The presence of non-background pixels indicates that 
y(— 4)]/[t(— 2>t(— 4)] the person has crossed the boundary of that rectangle. 

The recent velocity here is 40 This is useful to simulate the situation where a person 

touches something on the screen. Alternatively, the 

. ! item in the box could move so as to simulate someone 

jo = N (wo 2 + vya 1 ) throwing a brick or other object 

While the invention has been described with refer- 
The system includes the ability to smooth information ence to the preferred embodiment, it will be appreciated 
in order to eliminate noise. However, with too much by those of ordinary skill in the art that various modifi- 
smoo thing, the response time of the system is noticeably cations can be made to the structure and function of the 
sluggish, and with too little smoothing, there is notice- invention without departing from the spirit and scope 
able shaking of the observed position of the hand due to — thereof, 
system noise. In order to overcome these two limita- I claim: 

tions, an adaptive smoothing process is employed. The 1. A virtual reality game method for use by at least 
smoothed hand position sx, sy, is detennined as follows: one human player who wears at least one identifiable 

object having a unique color, said game comprising the 
55 steps of: 

M (a) scanning a background and creating a first color 

* = N (rar 2 + fp 2 ) table of background colors; 

(b) scanning said identifiable object; 
where vx-[x(0>x(-2)]/[t(0>-t(-2)]and vy=[y(0>- (c) creating a second color table with colors distino 
y(-2)J/[t(0)-t(-2)] $0 tiv e only to said identifiable object by deleting all 

Case 1: v is small (Le. v<v0 for some threshold colors that said identifiable object has in common 

vO) — smooth a lot so that jitter will not be seen with said background; 

sx=[x(0)+2*x(— l)-f x(— 2)]/4 (d) repeatedly scanning said player wearing said iden- 

sy=[y(0)-f 2*y(- i)+ y (_2)]/4 tifiable object against said background and creating 

Case 2: v is moderate (Le. v0<=v<vl for some 65 frames of pixels in a frame buffer representative of 
threshold vl) — do moderate smoothing said colors in said scan; 

sx=[x(0)+x(-l)]/2 (e) identifying said colors in said frames that are 

sy=[y(0)+y(-l)]/2 unique to said identifiable objects; 
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of: 



(0 predicting an area, less than a full frame^' where 
said identifiable object will be located in subse- 
quent frames; 

(g) scanning said predicted area to locate said identifi- 
able object; and, S 

(h) using said identifiable object to manipulate a vir- 
tual object toward a virtual goal 

2. The method of claim 1 further comprising the step 



18 



locating the centroid of said identifiable object in 10 
order to establish specific x and y coordinates for 
said identifiable object 

3. The method of claim 2 further comprising the steps 
of: 

(j) attaching said virtual object to said identifiable IS 
object in a display; 

(k) detecting the movement of said identifiable object 
to distinguish between at least two different ges- 
tures involving said identifiable object; and, 

(1) propelling said virtual object toward said goal in 20 
response to the recognition of one of said at least 
two gestures made by said identifiable object 

4. The method of claim 3 wherein said identifiable 
object comprises a glove and said virtual object com- 
prises a ball and wherein said method further comprises 25 
the step of: 

(m) determining the direction and velocity of said 
gesture detected in step (k) and crediting the player 
with a score if said direction and velocity are sub- 
stantially in the direction of said goal. 30 

5. The method of claim 4 further comprising the steps 
of: 

(n) determining the height of said player by detecting 
when a scan of said player against said background 
first comes into contact with a non-background 35 
object; and, 

(o) automatically scaling the height of a virtual oppo- 
nent in proportion to the height of said player. 

6. The method of claim 5 further comprising the step 
of: 40 

(p) performing a coarse-to-fine search by searching 
only a limited number of lines of said predicted 
area from step (g) first, and then increasing the 
number of lines searched untfl said glove is located. 

7. The method of claim 6 further comprising the step. 45 
of: 

(q) increasing the size of said predicted search area to 
a larger search area if said glove is not located. 

8. The method of claim 7 further comprising the step 
of: 50 

(r) searching the entire frame stored in said frame 
buffer, beginning with the center thereof, if said 
glove is not located in said predicted search area 
detennined by step (g) or in said larger search area 
as determined in step (q). 55 

9. The method of claim 8 further comprising the step 
of: 



(s) mcreasing the population of identifiable object 
color specific pixels by merging a first table of 
possible colors with a second master table of glove 
colors and then increasing the merged table of 
glove colors by filling in the voids surrounded by 
glove colors and by allowing neighboring colors 
on the edges of said glove colors in the merged 
table to take on glove specific color values. 

10. The method of claim. 9 further comprising the 
step of: 

(t) organizing pixels in said color tables by chroma X, 
chroma Y and luma, wherein each pixel is format- 
ted initially by two bits of chroma X, followed by 
two bits of chroma Y, and followed by six bits of 
luma, 

wherein said color values in said color tables are first 
compared using said above information as a means 
of rapid comparison of said values. 

11. The method of claim 10 further comprising the 
step of: 

(u) determining the width of said player so that said 
player's location can interact with a background. 

12. The method of claim 10 further comprising the 
step of: 

(v) deterniining the outline of said player so that said 
player can interact with a background. 

13. The method of claim 3 wherein said game in- 
cludes two players each wearing different identifiable 
object gloves, and wherein said two players play against 
each other to put said virtual object in said virtual goal. 

14. The method of claim 3 wherein predetermined 
gestures made by said identifiable object cause the 
image of said player to move with respect to said dis- 
play, 

wherein the movement of said player causes the 
player to appear as though the player was in an 
environment having gravity substantially smaller 
than that of the earth. 

15. A virtual reality game apparatus for use by at least 
one player having on his person an identifiable object 
having a distinctive color, said apparatus comprising: 

scanning means for scanning said player and for stor- 
ing said information in a frame buffer; 

color identifying means for deterrnining which of said 
colors of said identifiable object is most accurately 
identifiable and, for locating said identifiable object 
from the information in said frame buffer based 
upon its distinctive color when said player is wear- 
ing said identifiable object; and, 

display means for displaying said player and said 
identifiable object having said most accurately 
identifiable color in a virtual reality setting, 

wherein said player is correctly positioned within 
said virtual reality setting based upon the location 
of said identifiable object with said most accurately 
identifiable color. 
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